Use https for dataset download URLs where the host supports TLS - #9631
Use https for dataset download URLs where the host supports TLS#9631fjankovi wants to merge 1 commit into
Conversation
Several dataset classes download over plain http. Where the host serves the same content over TLS, switch to https: Food101 data.vision.ee.ethz.ch STL10 ai.stanford.edu SEMEION archive.ics.uci.edu MovingMNIST www.cs.toronto.edu Places365 data.csail.mit.edu KMNIST codh.rois.ac.jp FashionMNIST is not a scheme swap: it pointed at the S3 *website* endpoint (fashion-mnist.s3-website.eu-central-1.amazonaws.com), which serves http only by design. Point it at the REST endpoint for the same bucket instead (fashion-mnist.s3.eu-central-1.amazonaws.com), which supports TLS. Every changed URL was fetched over https and checked against the checksum already pinned in the source. The four small datasets match exactly (FashionMNIST 4/4, KMNIST 4/4, SEMEION, Places365 devkits 2/2); the large archives were confirmed by request. The remaining http URLs are deliberately left alone, with a comment added where the reason is not obvious: yann.lecun.com refuses connections on port 443 (and now 404s) ufldl.stanford.edu certificate is valid only for ai.stanford.edu shuoyang1213.me certificate is for *.github.com Rewriting those would turn a working download into a hostname-verification failure. LFW, PhotoTour and SUN397 are left untouched as well; those hosts no longer resolve or 404 on both schemes, which is tracked separately in pytorch#9580, pytorch#8711, pytorch#9002 and pytorch#9348. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/9631
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
Hi @fjankovi! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
1 participant
Several dataset classes download over plain
http://. Where the host serves the same bytes over TLS, this switches them tohttps://.Changed
data.vision.ee.ethz.ch200, fullContent-Lengthai.stanford.edu200, fullContent-Lengtharchive.ics.uci.educb545d37…www.cs.toronto.edu200, fullContent-Lengthdata.csail.mit.educodh.rois.ac.jpFashionMNIST is not a scheme swap. It pointed at the S3 website endpoint (
fashion-mnist.s3-website.eu-central-1.amazonaws.com), which is HTTP-only by design — S3 website endpoints do not support TLS. This points it at the REST endpoint for the same bucket (fashion-mnist.s3.eu-central-1.amazonaws.com), which does.Every changed URL was fetched over https and checked against the checksum already pinned in the source. The small ones match exactly; the multi-GB archives were confirmed by request rather than downloaded in full.
Deliberately not changed
Three hosts cannot serve https, and rewriting them turns a working download into a hostname-verification failure. A short comment now records why, so the next sweep doesn't complete them:
yann.lecun.comufldl.stanford.eduai.stanford.edu(SAN mismatch)shuoyang1213.me*.github.com(SAN mismatch)All three remain checksummed. Note MNIST's
yann.lecun.comentry is the fallback mirror —https://ossci-datasets.s3.amazonaws.com/mnist/is tried first — and it now returns404for the four resource files regardless of scheme, sotest_url_is_accessible[MNIST, …]already fails onmain. That's pre-existing and out of scope here; happy to open a separate issue.LFW, PhotoTour and SUN397 are also left alone: those hosts no longer resolve (
vis-www.cs.umass.eduandicvl.ee.ic.ac.ukare NXDOMAIN) or 404 on both schemes. Already tracked in #9580, #8711, #9002 and #9348.Testing
test/test_datasets_download.py, run against the patched tree:MovingMNIST and Food101 have no fixture in that file, so they were verified by direct request.
No behaviour change beyond the URLs; all checksums are untouched.